Surgical stereo vision systems and methods for microsurgery

ABSTRACT

Surgical stereo vision systems and methods for microsurgery are described that enable hand-eye collocation, high resolution, and a large field of view. A digital stereo microscope apparatus, an operating system with a digital stereo microscope, and a method are described using a display unit located over an area of interest such that a human operator places hands, tools, or a combination thereof in the area of interest and views a magnified and augmented live stereo view of the area interest with eyes of the human operator substantially collocated with the hands of the human operator.

CROSS-REFERENCE TO RELATED APPLICATION(S)

The present non-provisional patent application claims priority to U.S.Provisional Patent Application Ser. No. 61/537,592 filed Sep. 21, 2011and entitled “A Surgical Stereo Vision System for Microsurgery,” thecontents of which are incorporated by reference herein.

FIELD OF THE INVENTION

Generally, the field of art of the present disclosure pertains tosurgical vision systems and methods, and more particularly, to surgicalstereo vision systems and methods for microsurgery that enable hand-eyecollocation, high resolution, and a large field of view.

BACKGROUND OF THE INVENTION

Micromanipulations such as microsurgery require a distortion freemagnified and binocular stereoscopic view of its operating field 101.Surgical Loupes, Surgical Light Microscope (SLM) and its digitaladaptation, namely, Digital Stereo Microscope (DSM) provide such viewsof the operating field 101. SLMs have a number of optical elements thatprovide magnification, working distance and distortion correction. SLMshave an objective lens for collecting the light from the subject, a pairof oculars for viewing the subject and a number of optical elementswhich enhances the view through magnification and correction ofdistortion. SLMs are improvements on the classical double barreledbinocular microscope to address issues of distortion, limited workingdistance, limited field of view, and brightness. An exemplary prior artSLM/DSM 100 is illustrated in FIG. 1. To increase magnification, eitheran objective lens 102 is changed or zoom lens system 104 is moved. Thepair of oculars provides views having the required parallax to the rightand left eye. SLMs provide instant stereo view for depth perception tothe human visual system. Operating fields 101 provided through visualaids such as SLMs, DSMs and surgical loupes should allow a user toperform bi-manual microsurgical manipulations with the subject/objectsin the operating field and optionally sharing the operating field withone or more collaborators.

FIG. 2 illustrates the relationship between distance, D, to an object200 from the lens 202, focal length, f, of the lens 202, separation, B,of the lenses 202 and the disparity |l−r| of the image pixels in theleft and right eye views in a stereoscopic view. Field of View (FoV),Depth of Field (DoF), Working Distance (W) and magnification (Z) areinter-related in an optical system. When magnification is increased, thefield of view and depth of field is decreased. Various conventionalsystems and methods have added optical elements to rectify theselimitations but the key dependency between FoV and Z remains. Addedoptical elements also increase the size of the microscope, opticalimpedance and reduce the brightness. Additional lighting innovationsfurther increase the size, but the main drawback of FoV and Z dependencestill remains.

Using the current microscopes, it is not possible to view a locationthat is not in the optical path through the objective lenses 102 and theoculars 120. Users will need to reorient the object or subject (example,patient) to obtain a view from another angle. In addition, physicalconstruction of the microscope allows very limited numbers of oculars.Optical microscopes do not allow additional content to be added to theview or enhance the view with mensuration or annotation.

Additionally, there are many usability issues in performing micromanipulations using SLMs. A user of a SLM gets visual fatigue becausethe exit pupil diameter of the ocular lens is very small and lateralmovements of the observer causes motion in the field of view. Viewingstereo through surgical microscopes is also a learned skill. It takessignificant amount of practice to be able to find the right combinationof eye muscle coordination and inter-papillary distance setting of themicroscope that allows viewing through both eye pieces at the same timeto result in a brain-fusible stereo pair projected on to the retina.This makes stereoscopic viewing through binocular microscope tedious andtiring. Each time, when the eyes are taken away from the binocular eyepieces and brought back; there is a certain degree of eye muscleadjustment to be made before the view becomes stereoscopic. Thisinconvenience adds to the operational time and operator fatigue.

Another limitation is the narrow field of view. Microscope constructionmakes use of a large number of optical elements arranged in a barrel.The large number of optical elements narrows the field of view. FoV getsfurther reduced as the magnification is increased. Narrow field causesthe surgeons to lose the visual context frequently. A typical correctiveaction is to zoom out to bring back the context and then zoom in toachieve magnification while maintaining the tool or tissue in the fieldof view. This too adds to the operational time and operator fatigue.Surgeons cooperate with human assistants who share the same workspace asthe surgeon. The assistant should view what the surgeon is viewing, inorder to be of operational assistance. Physical construction of SLMmicroscopes typically allow only up to one assistant. In addition, eachsurgeon should have independent control of viewing the samefield-of-view by controlling the lighting, contrast and magnification.

Many surgeries last hours and the fixed posture (looking through the eyepieces) contributes significantly to the fatigue experienced bysurgeons. In a lengthy surgery, multiple surgeons and assistants maytime multiplex. Individual calibration of the microscope should be doneto get continuity. Though magnified view is a significant surgical aid,due to the above limitations only very few surgeons are able to performmicro surgery, the surgery that uses microscopes, though there are farmore surgeons with excellent surgical skills.

Referring back to FIG. 1, conventional SLMs/DSMs 100 use high resolutionand speed imaging sensors such as (Charge Coupled Devices) CCDs 110 tosupplement the oculars. The SLM/DSM 100 is retrofitted with imagingsensors at an eye piece 120 and then the view is sent to a stereoscopicmonitor 112 via a display processor 114 to view the FoV of an object120. Specifically, the CCDs 110 can receive the view from beam splitters122 before the eye piece 120. Surgeon uses stereoscopic viewingtechniques such as passive stereo glasses, active stereo shutter glassesand auto stereoscopic displays. Since the imagery is acquired anddisplayed digitally, these microscopes are called digital stereomicroscopes (DSM). A commercial example is the digital stereo microscopeby TrueVision™. When using the TrueVision™ microscope, surgeon caneither look through binocular barrels of the optical microscope as inthe case of the traditional microscope or look away at the stereoscopicscreen to view the workspace. The disadvantages of the former approachhave already been discussed. In the latter viewing setting, as thedisplay 112 is located elsewhere, surgeon loses the key hand-eyecollocation and as a result compromises the hand-eye coordination neededfor highly dexterous manipulations under magnified view. In addition,since the basic field of view is captured using the objectives of atraditional SLM, it suffers many of the limitations of the SLM describedearlier.

The human visual system has a variable resolution vision. The resolutiondrops from the foveal vision to the peripheral vision. The visual acuitydeteriorates from central (−1, +1) degree visual field to the (−59,+110) degree visual field. Roughly, the (−2, +2) degree visual field hashalf the resolution of the (−1, +1) degree visual field, (−5, +5) degreevisual field has half the resolution of the (−2, +2) visual field and(−12, +12) has half the resolution of the (−5, +5) visual field.However, SLMs or DSMs do not provide any visual aid to match the visualresolution variance. Performing such variable resolution without depthdistortion effects is also a challenge.

Stereo microscopes without oculars are another conventional system.These use mirrors that rotate fast to project the right and left view toright and left eye without having an eye piece. A user of thismicroscope needs to place the users eyes aligned to the projected spaceto be able to see stereo. While the above system affords some freedom ofhead movement, the projected space is quite narrow and it is easy toslip out and loose stereo vision. This system also requires training touse and even after training, at each usage there is a task of aligningeyes to the projected space. Peripheral vision can easily distract theeyes from seeing stereo. These microscopes have not found a home inoperating theatres yet and are used in the manufacturing industry forinspection of printed circuit boards and other miniature electronicassemblies.

BRIEF SUMMARY OF THE INVENTION

In an exemplary embodiment, a digital stereo microscope apparatusincludes a display unit located over an area of interest such that ahuman operator places hands and/or tools in the area of interest andviews a magnified and augmented live stereo view of the area interestwith eyes of the human operator having a view substantially collocatedwith the hands and/or tools of the human operator; an image displaylocated on a first side of the display unit providing the magnified andaugmented live stereo view; an optical system and an image acquisitionsystem located in part on a second side of the display unit providingimage capture and data capture of the area of interest; and a processingsystem communicatively coupled to the image acquisition system and theimage display providing the magnified and augmented live stereo view tothe image display based on the image capture and the data capture of thearea of interest from the optical system and the image acquisitionsystem, wherein the optical system and the image acquisition system isconfigured to provide the image capture with adjustments to provide themagnified and augmented live stereo view performed by the processingsystem.

In another exemplary embodiment, an operating system with a digitalstereo microscope includes an articulated arm comprising a displaymounting system; a display unit connected to the display mountingsystem; an operating site over which the display unit is located suchthat a human operator places hands and/or tools in the operating siteand views a magnified and augmented live stereo view of the operatingsite with eyes of the human operator having a view substantiallycollocated with the hands and/or tools of the human operator; an imagedisplay located on a first side of the display unit providing themagnified and augmented live stereo view; an optical system and an imageacquisition system located on a second side of the display unitproviding image capture and data capture of the operating site; and aprocessing system communicatively coupled to the image acquisitionsystem and the image display providing the magnified and augmented livestereo view to the image display based on the image capture and the datacapture of the operating site from the optical system and the imageacquisition system.

In yet another exemplary embodiment, a method using a digital stereomicroscope includes positioning and adjusting a display unit above anoperating site; enabling the display unit, wherein the display unitcomprises an optical system and an image acquisition system located on aside of the display unit adjacent to the operating site, and wherein thedisplay unit comprises an image display on a side of the display unitopposite to the side facing the operating site; providing image captureand data capture of the area of interest via the optical system and theimage acquisition system; processing the image capture and the datacapture via a processing system; positioning a user's hands and/or toolsin the operating site while maintaining the user's eyes having a view ina collocated manner looking at the image display; and presenting amagnified and augmented live stereo view of the operating site via theimage display based on the processed image capture and the processeddata capture of the area of interest from the optical system and theimage acquisition system.

BRIEF DESCRIPTION OF THE DRAWING(S)

Exemplary and non-limiting embodiments of the present disclosure areillustrated and described herein with reference to various drawings, inwhich like reference numbers denote like method steps and/or systemcomponents, respectively, and in which:

FIG. 1 describes a prior art SLM/DSM system with optional digitaladaptation by replacing the oculars by a pair of image sensors andstereoscopic display;

FIG. 2 describes the prior art relationship between distance to theobject, focal length of the lens, separation of lenses and the disparityof the image pixels in the left and right eye views in a stereoscopicview;

FIG. 3 describes a digital stereo microscope with image augmentationcapabilities according to an exemplary embodiment;

FIG. 4A describes one preferred arrangement of image sensors in theback-side of the display unit according to an exemplary embodiment;

FIG. 4B describes another preferred arrangement of image sensors on aflexible arm fixed to a stand according to an exemplary embodiment;

FIGS. 5A-5B describes a hand-eye collocated stereo display with FIG. 5AFIG. 5. a describing a schematic of the system with two users sharing adisplay and FIG. 5B describing the display arrangement and indirectviewing according to an exemplary embodiment;

FIG. 6 describes a multi-camera system achieving high magnification andhigh resolution using an optical lens assembly, an image sensor, adisplay frame processor and a flat panel display according to anexemplary embodiment; and

FIG. 7 describes the user interface for configuring the digitalmicroscope.

DETAILED DESCRIPTION OF THE INVENTION

In various exemplary embodiments, surgical stereo vision systems andmethods for microsurgery enable hand-eye collocation, high resolution,and a large field of view. Multiple multimodality sensors, includingimage sensors, capture properties of an object under the view of thesystem and the user's micro manipulations of the object. The capturedlive view is augmented with data from sensors and external databases toaid real-time micro manipulation of the object. The system providesmultiple views on one or more flat-panel stereoscopic displays,collaborative manipulation of the object, real time measurements, andpanning and tilting of the field of view without moving the object. Thesystem supports microsurgical manipulations to be performed on a patientby a group of surgeons and imparts training to surgeons by recording andreplaying the surgical manipulations with a phantom object in the fieldof view. The system includes a DSM that provides augmented andcollocated live stereo views to rectify the limitations of conventionalsystems. The system uses optical elements and imaging sensors for imagecapture, digital computing for processing, and digital displays forstereoscopic viewing. In addition to the optical imaging, the systemincludes multimodality sensors and different image augmentation schemesfor improving the visualization of and interaction with the operatedsite.

An objective of the surgical stereo vision systems and methods is toachieve SLM parity in resolution and magnification used in surgery. Thisincludes achieving continuous zoom using a movable intermediate lensassembly between the objective lens and ocular without the use ofintermediate lens assembly.

Another objective includes a reduction in viewing latency to beunnoticeable to the human eye. Due to the indirect viewing of the realscene, latency introduced between what happens at an operating site andwhat is seen in the video of the live interaction which may disrupt thesurgical procedure.

Yet another objective is to provide better functionality thanconventional SLMs and DSMs by providing scene augmentation and alternateviews. Multimodality sensors and image sensors on flexible arm can beused to provide augmentation and alternate views. The video streams arestored and replayed to allow surgical training in the same environmentas that of the actual surgery.

Another objective of the surgical stereo vision systems and methods isto provide better ergonomics of stereo interaction for long surgeriesthrough a hand-eye collocated stereoscopic display with support for pan,variable zoom, and tilt of the field-of-view. Advantageously, astereoscopic view will not be lost by moving the head. Still yet anotherobjective is to allow collaboration in the surgery having multiplesurgeons and surgery assistants to simultaneously share the live viewand multiple surgeons to cooperate sequentially without causingstereopsis related delays.

Referring to FIG. 3, in an exemplary embodiment, a block diagramillustrates a digital stereo microscope system 300. The digital stereomicroscope system 300 includes an optical system 302, an imageacquisition system 304, a processing system 306, and an image displaysystem 308. The image acquisition system 304 includes a collection ofobject property sensors 312, a collection of lighting resources 314, anda collection of image sensors 316, in data communication with the imagedisplay system 308 via the processing system 306 and connected to theoptical system 302 through the image acquisition system 304. The opticalsystem 302 captures the field of view and sends an optical image to theimage sensors 316 for generation of high resolution digital image of thescene. The image sensors 316 can include, without limitation, CMOS, CCD,photo-voltaic sensors, infra-red sensing elements, and the like.

The image processing sub-system 320 includes signal data processors 321,control data processors 322 and image data processors 325. The humanoperator 350 places the display units 341 and sensing elements 313 toyield a stereo view of the operating site using the display control 323.The lighting resources 314 are configured to provide variable amounts ofvisible light to the field of view. The object property sensors 312 areused to provide data to the processing system 306. The object propertysensors 312 can include, without limitation, infra-red transceivers,ultra-sound transceivers, laser transceivers, light field cameras, depthsensing cameras, and the like.

Referring to FIGS. 4A-4B, in exemplary embodiments, a diagramillustrates a display/image sensor unit 400. In particular in FIG. 4A,components of the optical assembly 302 and the image acquisition system304 can be disposed on a back side of the unit 400 with a display unit320 on the front side of the unit 400. Further, each of the varioussensing elements 312, 314 can be independently or collectivelycontrolled by a human operator 330. As is described herein, the designof the display/image sensor unit 400 is critical to enabling hand-eyecollocation as the display 320 is placed over an object of interestwhile hands of the human operator 330 manipulate the object of interest.The sensing elements 312, 314 may either emit sensing rays such asvisible light, ultrasound or infra-red, collect sensing rays, or bothemit and collect sensing rays.

Some of the sensing elements 312, 314 may be placed on a flexible arm420 as shown in FIG. 4B and brought to an operating site. The operatingsite includes the field of view of the system 300 and outside the fieldof view of the system 300. Some of the sensing elements 312, 314 such asthe visible light camera and infra-red camera may be placed on both theflexible arm 420 and the back side of the display unit 400. Thearrangement of camera elements and other sensors may be along a curve.The arrangements can also be along a line or along a curved surface, soas to aid increasing the field of view. The position and orientation ofone or more sensing elements may be controlled by the human operator.Also, the image sensors 316, the light sources 314, and/or the objectproperty sensors 312 can be arranged on one of the flexible arms 420which can be moved to focus to the operating site with manualassistance. These additional sensing elements on the flexible arms 420are used to obtain views that are not possible with zooming and panningof the current field of view. For example, the self-occluding portion ofa tissue can be made visible only through an alternate angle of view. Itrequires either moving the patient or tilting the image sensors 316fixed on the back-side of the unit 400. By tilting the camera, thecurrent view of the operating site is lost, i.e. hand-eye collocation islost with the unit 400. By having the additional image sensors 316 fixedto the flexible arms 420, alternate views can be composed within thecurrent viewing context. Alternate views can also be obtained from theplurality of image sensors 316 arranged at the back of the display. Thelimited view of the DSM and SLM systems are overcome through thisarrangement.

Referring to FIGS. 5A-5B, in exemplary embodiments, diagrams illustrateuse of the unit in a surgery system 500 with a hand-eye collocatedstereo display. FIG. 5A shows a schematic of the system 500 with twousers sharing the system 500 and FIG. 5B shows the display arrangementand indirect viewing of the display unit 400. Specifically, a humanoperator 330 initially places the display unit 400 along with themounted sensing elements above an operating site 502 to view theoperating site 502. An anatomical view of the operating site 502 isprojected onto the display unit 400 and the human operator 330 can givecontrol commands to zoom, pan, focus, etc. The operating site 502 can bean area of interest over which the display unit 400 is located and whichis magnified in the display 320 of the display unit 400.

The processing system 306 generates commands to orient one or more ofthe sensing elements 312, 314, 316 in response to the human operator's330 commands. The human operator 330 may adjust the height and viewingangle of the display unit 400 without invoking the processing system306. For example, the system 500 can include an articulated arm 504 anda display mounting system 506 that allow adjusting the position of thedisplay 400 at convenient viewing height and angle as well as leavingsufficient working distance. Based on the human operator's 330 viewingand working distance configuration, and the indication of the operatingsite 502, the selection of elements 312, 314, 316 and its orientation isdecided by the processing system 306.

The location of the operating site 502 may be indicated by projecting acursor 510 onto the scene. The human operator 330 moves the cursor tothe desired site. The location of the operating site 502 may also beindicated by the human operator 330 by entering the anatomical locationof the operating site 502 and then perform operations of pan, tilt andzoom to select the view and desired magnification. The human operator330 performs the pan, tilt and zoom using one or more of the followingmethods. The human operator 330 may use a control knob or knobsassociated with the unit 400 to increase and decrease each of theseparameters. The human operator 330 may use a virtual object projected onthe scene to control the parameters. Multiple methods are provided forconvenience and safety. The human operator 330 may adjust the disparityand the distance between the eye and the display 320 to obtain thestereo view. These finer adjustments are done using the tool tip orusing the control widgets.

The system 300, 500 may be operated in three modes, the pre-operativemode where manual control as well as control knobs is used to positionthe system 300, 500. During the surgical operation mode, the finecontrols are done using the virtual objects such as the virtual cursor,and control widgets projected onto the field of view. In thepost-operative mode, a greater field of view is provided to track thetools returning to the tool chests.

The image sensors 316 can include various cameras which can bepositioned in various locations in the system 300, the display unit 400,and the surgery system 500. In an exemplary embodiment, the surgicalvision systems and methods operate with the various cameras eachacquiring images of an area of interest at a maximum magnification andthe associated reduction in magnification being done in software via theprocessing system 306. Of course, the surgical vision systems andmethods can also rely on lens associated with the various cameras. In anexemplary embodiment, some of the cameras are fixed on the back side ofthe display unit 400. In another exemplary embodiment, some of thecameras are configured to move, pan, tilt, etc. In yet another exemplaryembodiment, some of the cameras are mounted in different locations inthe surgery system 500 from the display unit, e.g. ceiling, walls, etc.In yet another exemplary embodiment, some of the cameras are movablyattached to the flexible arms 420. In still yet another exemplaryembodiment, some of the cameras can be in a linear or curved cameraarray. Of course, combinations are contemplated of the foregoing. In thevarious exemplary embodiments, camera position is such that a view canbe obtained of the area of interest with the operator's 330 hands and/ortools in between the display unit 400 and the area of interest, i.e. theoperating site 502. Importantly, the cameras are all communicativelycoupled to the processing system 306 for presenting live streams whichcan be manipulated by the processing system 306 for display to manydifferent users simultaneously including different views to differentusers simultaneously. The cameras can be collectively positioned suchthat the human operator 330 can maintain hand-eye collocation with theoperating site.

Referring back to FIG. 3, in an exemplary embodiment, the system 300uses two cameras (i.e., image sensors 316) mounted on the back of thedisplay unit 400, a camera (i.e., an image sensor 316) mounted on theflexible arm 420, a processing unit collectively referred to as theprocessing system 306, and two display units 400 connected to theprocessing system 306. The processing system 306 can have severalhundred processors for processing the camera generated live stream.Specifically, the processing system 306 is configured to receive datafrom the image acquisition system 304 and generate stereoscopic views onthe image display system 308 for hand-eye collocation at the operatingsite 502. Stereoscopic views include one exemplary presentation on theimage display system 308. Others are also contemplated for providingdepth perception. While presenting stereoscopic views, the humanoperator 330 can use glasses for viewing the image display system 308.An exemplary description of a system using shutter glasses is describedin U.S. Patent Publication No. 2010/0103247, co-invented by the inventorof the present application, published Apr. 29, 2010, and entitled “ANIMAGING DEVICE AND METHOD. Since this prior-art system uses cameraconfiguration to generate stereo pairs, it has same physical limitationsas that of the optical microscope that the field-of-view, depth-of-fieldand magnification are interrelated. The prior-art system gets 2 cm to 8cm work space which is insufficient to perform surgical manipulations.The present invention eliminates those limitations.

The processing system 306 can include various processors such as signalprocessors 340, control processors 342, display control 344, storageprocessors 346, data processors 348, and display processors 350.Variously, the processing system 306 is configured to processconfiguration and control commands, perform storage and retrieval ofobject properties such as via an external object property system 352 aswell as for processing the object property sensor data from the propertysensors 312. The signal processors 340 communicate and control the imageacquisition system 304. The control processors 342 communicate andinterface with the display control 344 which communicates and interfaceswith the human operator 330.

In an exemplary embodiment, the control processors 342, the displaycontrol 344, the signal processors 340, and the storage processors 346resides in a host central processor unit or units (CPU) and the dataprocessors 348 resides in a large number of processors attached to theCPU through an interface, i.e. a bus such as Peripheral ComponentInterconnect Express (PCIe). The communication between host CPU and thePCIe hosted processors can be very infrequent. In configurations where alight projector for the lighting 314 is used to augment the display 329,the host CPU provides the display signal to the projector. The system300 can also include multiple display processors 350 connected to thedata processors 348 either via the interface or a network 354connection. The display processors 350 are also communicatively coupledto the display 320.

For initial configuration, the system 300 is powered on and the displayunits 400 are moved to the site of the surgery. Once stereo views arevisible in the display 320, the human operator 330 picks sensors fromthe set of sensors presented on the display 320 and places them in thethree dimensional (3D) space. When sensors are connected, the data willbe displayed at those positions. Once the human operator 330 viewoperations are completed, the sensor icons are removed from the display320 by user interaction.

Once the first display unit 320 is calibrated to convenience, the seconddisplay unit 320 is calibrated by another respective user to suit theview parameters. The disparity for the stereo pairs may be different forthe second user; however, the camera movements are now arrested. Theuser operation results in building disparity computation matrix thataffects only the processing of the stereo pairs. The relative cameramovements suggested by the second user is factored into a processingvector which is applied to the incoming viewing pairs. When both cameraunits are calibrated, further fine tuning of the calibrations ispossible with both display adjustments results in changes in atransformation vector. This can be repeated for additional users.

Multiple surgeons and assistants may calibrate the system to theirvisual comfort and store the calibration against their named icons. Thenduring the surgery, the respective surgeon can bring the calibration bytouching and activating their stored configuration parameters. Surgeonscould also be recognized by the system through biometric recognition.The transformation vector may be either stored in the local displayprocessor 350 or in the processors 348). When the transformation vectoris stored in the local display processor 350, the processors 348 fusethe information that need to be processed by the human visual system.Since the various cameras associated with the system 300, the unit 400,and the system 500 are all capturing portions of the area of interest,the processing system 306 can be used to synthesize different live viewssimultaneously for different displays 320. For each view the operatormay choose to see the same as what the other operator is viewing or anindependent view (through another set of camera) or a combination of theviews on the same screen. Generally, the system 300, 500 is configuredvia the various components contained therein to capture a whole areafrom an image perspective and to synthesize each person's view on thefly via the processing system 306.

With respect to hand-eye collocation, the human operator 330 performinga surgery at the operating site 502 will see the display 320 as if thedisplay 320 was a magnifying transparent glass without the limitationsof a magnifying glass. In a basic case of no magnification, it would belike looking through a glass. Of course, the display unit 320contemplates magnification and augmentation of the magnifying glass.However, the systems and methods described herein are not limited to acollocated view. The operator 330 can choose to view an area of interestthat would not be visible if he were to look through the glass model,for example the side of the work area facing away from the operator. Thesystem 300, 500 could have cameras arranged in such a way to image thoseand make them available to the operator without tilting the scope or thesubject. This is a huge advantage in surgery. Even with the directcollocated view, there will always be occlusion due to the operator handand tools hiding the work area. Once again cameras can be placed(multiple statically placed cameras on a plane or a curved surface orfew cameras that are moved electronically using motors) such that theocclusion can be minimized.

Composing the signal information from the image acquisition system 304with the live stream displayed on the display system 308 involves atwo-step process. In a first step, the information is streamed into theprocessing system 306 including device memory and processors that areresponsible for the portion of live stream data indexes the signalinformation and fuse them into the common display. A second stepincludes performing various processing techniques bon the information bythe processing system 306 prior to sending the processed information tothe display 320.

The system 300 can be controlled by contact and non-contact means. Tooltip position and gesture are used to enter into command mode. Forexample, during a surgery, surgeon needs to change the magnification orselect an opposite view. Surgeon can move the tooltip to a plasticreference object placed in the field of view of the microscope andhaving a specific pattern and touches it. This switches the system 300to command mode and brings up virtual widgets which can then be chosenbased on gestures. Magnification of widget overlay is controlled byup/down movements of the tooltip in command mode. Similarly, for cameraselection, the widget brings up a palette of cameras and user selectsthem by movement. The tooltip and the virtual object must be easilyrecognized from video stream and should be prioritized in the dataprocessing.

Another control operation is that a user annotates an object in thevisual field using a tool tip for sharing information for future use toaid a collaborating user. The annotation may also may be an instructionto the collaborating user and used immediately. For example, an operatoris able to measure angles and distances in the visual field betweenidentified points and identified lines.

In another control operation, the user may pan or tilt the field of viewor zoom a selected sub-field of the field of view. For example, asurgeon may want to see a portion of the current operating site occludedby a tissue. An alternate view can be requested without moving thepatient. In another example, the surgeon may mark a portion of thetissue in the command mode and request it to be zoomed. The zoomedportion may be displayed as an overlaid or as a separate a picture in adesignated portion of the screen.

The system 300 contemplates various methods for command input by thehuman operator 330, such as hand interaction, foot pedals, etc. Also,gestures are contemplated in the area of interest, i.e. by theoperator's 330 hand or tools. For example, the operator 330 can useordinary surgical tools for issuing commands to the system 300 inaddition to the foot pedal based control or the like. This could be doneby say first touching a special unique object in the field of view andthereafter using tool gestures. The operator 330 returns to normal modeby once again touching the special object.

In an exemplary embodiment, the display 320 can provide a highdefinition stereoscopic display of the field of view, and can include a1920×1080 pixel stereoscopic display. The display 320 can also displayadditional special elements. For example, the display 320 may showcontrol widgets in a designated portion of the display 320 and movementsin this portion of the display is treated as commands. The display 320may also contain object properties as detected by the property sensors312. For example, the temperature at an operating site may be displayedin a separate area. It may also be displayed as an overlay.

The display 320 of one user may be different from the display 320 ofanother user. For example, a user may choose to perform pan, tilt andzoom and see a different display than the collaborator in a specialdisplay mode. In the normal display mode, all views are shared.

In an exemplary embodiment, the zoom levels available in the system 300are from 6× to 12× though only a magnification of up to 5× is usedcommonly in surgery due to the dexterity limits of human hand. Highermagnifications are useful for robot guided surgery or otherapplications. The system 300 contemplates use in manual surgery, andmanual surgery with tremor reducing devices which limit the operablemagnifications.

Another type of magnification occurs when the system 300 uses aComplementary metal-oxide-semiconductor (CMOS)/charge-coupled device(CCD) sensor. The CMOS/CCD sensor can be 5 cm×5 cm in dimension with1920×1080 pixel read out for display on a 22 inch screen of 1920×1080pixels. The scale factor is of the order of 5 without using opticalzoom. In addition, optical zoom may be employed to increase themagnification of selected tissue sections for surgeon's view.

The system 300 overcomes many data processing and transmissionchallenges in conventional systems. For example, the system 300constructs a stereo view and let the human visual system compose a 3Dobject through stereopsis. A very high resolution of the live imagestream is necessary for faithful reproduction of the 3D object in finedetails. Especially, in microsurgery, where surgical manipulations aredone on a highly magnified object, the distortions are minimized to gainfine details of the operating site 502. The resolution requirementincreases with the magnification required. For a magnification of 3×,the high definition (HD) resolution (1920×1080 with 24 bits deep) is theminimum required. The number of frames from the camera sensors can be at30 frames/sec, giving rise to 30×1920×1080×3 bytes, which isapproximately 178 MB˜200 MB per second per sensor. With an average offour camera sensors active at the same time, 800 MB/sec bandwidth isnecessary. The high bandwidth requirement causes many challenges. Thenumber of memory transfers that are permitted in a frame operation mustbe limited to avoid causing delays.

For example, if the memory bus is 512 bits wide and memory clock is 1017MHz and are using double data rate RAM, then the peak theoretical memorythroughput is 1017×106×512×2/(8×10243)˜130 GB/sec. The theoreticaltransfer rate is not achieved, because it assumes a single memorytransaction with negligible setup and terminates costs. It only servesas a guide. In practice different elements of the memory segments areaccessed and based on the access pattern, the number of memorytransactions needed is much higher. Hence algorithms that process toproduce stereo pairs are used to have aligned memory access for thedevice to reduce the number of memory transactions.

Another problem is the computation needed to perform operations on thestereo pairs. Stereo pairs are produced by two cameras, each of focallength f, fixed on a baseline with a baseline distance B apart andcollects incident light on an object placed at a distance D. The twocameras will produce the image pixel which differs by a distance d forthe point object.

$\begin{matrix}{D = \frac{Bf}{d}} & (1)\end{matrix}$

The point object produced two image points, one for the left image andanother for the right image. Their difference in their position in theimage is an indication of the relative depth of the object. Finding thecorresponding points in each of the images and thereby finding thedisparity or the relative horizontal shift is essentially the stereocomputation problem. The problem is solved by finding the pixel withleast difference in intensity. This is an extensive computationrequiring large CPU resources and high memory bandwidth.

Typically, find the SSD (sum of squared differences) between the rightand left image intensities to identify the pixel correspondence in thepixel arrays L and R corresponding to the left and right images. Foreach pixel, the minimum SSD value indicates a candidate correspondencepair. The computation is nearly impractical to be done at live streamsat high resolutions such as HD resolution.SSD_(x,y)=Σ_(i=x+w) ^(x+w)Σ_(j=y−h) ^(y+h)(L[i,j]−R[i−k,j])²  (2)

If the pixels are to be corresponded, then the stereo images must besearched for sum-of squared distances. In addition, differences in thefocal length of the two cameras, the lighting, can all contribute toambiguity in determining correspondence in the image plane. For livestreaming of stereo, the difficulty is in performing the processing inreal time. The present system 300 uses selective processing of regionsto avoid delay. The regions that need to be updated more frequently areprocessed by more number of processors in the processing system 306.Specifically, the selective processing can include uniquely patternedobjects in the subject area (e.g., patterned clothes, reference frames,tools, etc.) to help match the two views. That is, the system 300 canmake use of the constraints associated with the operating site 503 foradditional information that can be used to make fast—uniquely patternedobjects in the subject area to help match the two views.

Also, the system 300 can include techniques that inflate the depthperception on a gradient with a focus point as its centre. The idea isto have greater clarity at the focal area and gradually decreasingclarity at areas further away from focus. This allows for speed andprocessing efficiency while focusing where it is important, the focalarea of the operating site 502.

The system 300 offers special operations such as removing occlusion ofthe tissue due to bleeding by selecting the IR camera sensor elementsand processes the streams originating from both IR and visible lightsensing elements.

As described herein, the processing system 306 can be divided into fivedata processing modules, the signal data processors 340, the controldata processors 342, the image data processor 348, the storageprocessors 346, and the display processors 350. The display control 344module is also part of the processing system 306. The signal dataprocessors 340 receive, via the elements 312, 314 m, non-image sensingelements of the digital microscope as well as vital signals from othersensing elements that are not part of the digital stereo microscope. Thereceived signals may indicate the current temperature of the operatingsite or pressure, pulse rate, ECG etc. These are monitored by functionalspecialists and the surgeon may be informed. When such information needssurgeon's attention, it can be brought to surgeons view by projectingthe information into operating view of the surgeon and away from theoperating site.

Information such as signal data, measurement data, and control widgetsmay be projected into the operating site by combining the images by theimage processor into a single stereo pair of images send to the displayunit. Alternatively, such information may be projected by a tinyprojector mounted on the back side of the display and the combined imagestream may be processed by the image data processor.

The system 300 can fuse a selected sensor stream of surgical motion andfuse it with the image of the surgical site to simulate conditions ofactual surgery by using the training surgical specimens mounted at theoperating site.

In addition to the camera and lighting elements mounted on the back sideof the display unit 400, a number of camera and lighting elements areprovided in a spoke-and-hub arrangement, mounted on the flexible arms420 to provide alternative views of the operating site 502. The imagedata processor 348 combines these newly presented views with the viewsfrom the camera elements mounted on the back-side of the display units400 to provide views from a different angle.

The operating surgeon is able to perform virtual tilting of the patientusing the views of the images produced by the spoke-and-hub camerasystem. The surgeon virtually tilt or roll the patient by using avirtual control widget which results in combining the views frommultiple of the spoke-and-hub camera system.

The image data processor 348 is able to compose sets of image stereopairs using multiple image views from the sensing elements both from thedisplay mounted sensing elements and from the spoke and hub sensingelements. The display unit 320 is optionally able to perform adjustmentsto the received stereo pairs when all display units are receiving samestereo pairs.

Registration and calibration of the system 300 can be performed beforethe surgery by using a phantom object to establish the relationshipbetween multiple sensor elements are their current positions. It mayalso be achieved with a real object or a marker placed in the field ofview. Along with position of the imaging, property sensing elements, thecorrespondence between the object coordinates and image coordinates isachieved through registration of a fixed marker in the field of view ofthe microscope.

The live video data, signal processing data, object property data, andcollaborating user's input are to be combined into the same virtualspace and presented in the stereoscopic display 320. There are differentprocessing steps. In a first step, the video data from multiple camerasare clipped for overlapping pairs of views and corrected for matchingdisparity. The disparity must match the XY zoom. The combining of otherdata is performed in one of the following ways. In one method, a lightprojector projects the data into the correct locations so that the liveview will have the combined data. In another method, the combining ofthe data is performed by the data processor 348.

The data processor 348 divides the entire data into predefined chunks.The definition of division may be based on auxiliary input signal suchas motion tacking, or may be based on static division such as gradationof the visual field. Division may also divide the visual field into theoperating field and command field. Some of the data are to be processedfaster than the other data. For example, the command field data must beprocessed before the operating field. Based on the nature of the data,each chunk of data is given a collection of processors.

Referring to FIG. 6, in an exemplary embodiment, a block diagramillustrates components of the system 300 interworking with the displayunit 400. The image acquisition system 304 can include CMOS/CCD sensors602 mounted on the display unit 400. The optical assembly 302 caninclude zoom lenses 604 and objective lenses 606 mounted on the displayunit. These components 602, 604, 606 can provide image data from theoperating site 502 to the data processors 348 which performs processingto form stereoscopic views which are provided to the display processors350 for display on the displays 320 on the display unit 400.

Referring to FIG. 7, in an exemplary embodiment, a screen diagramillustrates a screen shot 700 of the display 320 of the system 300. Asdescribed herein, the screen shot 700 can include a stereo view 702 ofthe operating site 502 as well as other data 704 which can includesensors and configuration data presented as widgets on the screen shot700.

There are several advantages to the system 300. Advantages are presentedfor the system 300 used in performing micro-surgery though similaradvantages will be present in other applications. Since the system 300allows hand-eye-collocation, both the gaze and hand movement arecoordinated and eye gaze is towards the hand movement, the surgeon'sstrain is reduced.

Large field of view is presented to the surgeon through the stereoscopicflat display panels that allows head movement without losing thestereoscopic view. The field of view could have a uniform resolution ora graded resolution. The large field of view takes away the need forconstant zoom-in-zoom-out needed to maintain context. The stereoscopicview is immediate and does not need training.

Since the system 300 does not require intensive training to performsurgical operations by looking through a microscope and adjusting themicroscope positions on the fly, it can be used in other surgeries wherethe surgeon is not trained in microsurgery. The system 300 improves thevisibility of the operating area. An example is surgery of the eyelidspracticed by cosmetic surgeons.

Another advantage is the display of critical patient information such asheart rate, which can be displayed either as direct numbers or as levelsalong with the same display unit or as a graph. This information may bedisplayed at a depth in the field of view of the surgeon, but withoutobstructing the view of the operating site. Other pre-surgery imagery ofthe operating site may also be overlaid with the live stream withcorrect alignment to aid in the surgery.

The live stream may be processed to provide guidance in terms ofmensuration and anatomical recognition. A Surgeon is able to change theview angle without moving the patient and is able to have views fromdifferent angles at the same time. The system 300 offers a platform forintegrating sensors and a method of combining the sensor informationwith the live video of the surgery. The hand movement of the surgeon andthe live video stream of the hand movement are displayed withoutnoticeable delay, the surgeon gets a feel of the real operating site asif the display units were not present. An expert can cooperate in thesurgical procedure from a remote site with some delay in the handmovements and its video stream and provide instructions to the operatingsurgeon.

The remote collaboration feature also offers teaching facility whereinthe expert surgeon conducting the operation can be mimicked by each ofthe training surgeons who have the display units with the remote stereoviews being integrated with the local movements. In such applications,the display unit is able to perform simple stereo image transformationsto combine the scenes by adjusting the camera positions and overlayingthe received images and local video. During operation, the surgeon isable to operate the system using virtual configuration objects. Thissatisfies the sterile requirement of the operating environments.Surgeons can use saved configurations to bring the stereo display totheir own preferred settings, when multiple surgeons time multiplexduring a long duration surgery.

It will be appreciated that some exemplary embodiments described hereinmay include one or more generic or specialized processors (“one or moreprocessors”) such as microprocessors, digital signal processors,customized processors, and field programmable gate arrays (FPGAs) andunique stored program instructions (including both software andfirmware) that control the one or more processors to implement, inconjunction with certain non-processor circuits, some, most, or all ofthe functions of the methods and/or systems described herein.Alternatively, some or all functions may be implemented by a statemachine that has no stored program instructions, or in one or moreapplication specific integrated circuits (ASICs), in which each functionor some combinations of certain of the functions are implemented ascustom logic. Of course, a combination of the aforementioned approachesmay be used. Moreover, some exemplary embodiments may be implemented asa non-transitory computer-readable storage medium having computerreadable code stored thereon for programming a computer, server,appliance, device, etc. each of which may include a processor to performmethods as described and claimed herein. Examples of suchcomputer-readable storage mediums include, but are not limited to, ahard disk, an optical storage device, a magnetic storage device, a ROM(Read Only Memory), a PROM (Programmable Read Only Memory), an EPROM(Erasable Programmable Read Only Memory), an EEPROM (ElectricallyErasable Programmable Read Only Memory), Flash memory, and the like.When stored in the non-transitory computer readable medium, software caninclude instructions executable by a processor that, in response to suchexecution, cause a processor or any other circuitry to perform a set ofoperations, steps, methods, processes, algorithms, etc.

Although the present disclosure has been illustrated and describedherein with reference to preferred embodiments and specific examplesthereof, it will be readily apparent to those of ordinary skill in theart that other embodiments and examples may perform similar functionsand/or achieve like results. All such equivalent embodiments andexamples are within the spirit and scope of the present disclosure andare intended to be covered by the following claims.

What is claimed is:
 1. A digital stereo microscope apparatus,comprising: a display unit fixed and located over an area of interestsuch that a human operator places hands, tools, or a combination thereofin the area of interest and views a magnified and augmented live stereoview of the area of interest with eyes of the human operator having aview substantially collocated and in a direct field and line of viewwith the hands, tools, or combination thereof of the human operator inthe area of interest, wherein the human operator is able perform headmovement without losing the live stereo view on the display unit; animage display located on a front side of the display unit relative tothe human operator providing the magnified and augmented live stereoview; an optical system and an image acquisition system located in parton a back side of the display unit providing image capture and datacapture of the area of interest; wherein the optical system and theimage acquisition system comprises at least one objective lens and atleast one image sensor; and a processing system communicatively coupledto the image acquisition system and the image display providing themagnified and augmented live stereo view to the image display based onthe image capture and the data capture of the area of interest from theoptical system and the image acquisition system, wherein the opticalsystem and the image acquisition system is configured to provide theimage capture with adjustments to provide the magnified and augmentedlive stereo view, in real-time, performed by the processing systemutilizing stereopsis.
 2. The digital stereo microscope apparatus ofclaim 1, wherein the optical system, the image acquisition system, andthe processing system provide continuous zoom without an intermediatelens assembly with the continuous zoom being performed by the processingsystem.
 3. The digital stereo microscope apparatus of claim 1, whereinthe optical system and the image acquisition system are configured tocapture image data at a maximum resolution of associated hardware withaugmentation, magnification, and reduction of resolution of the imagedata being performed by the processing system.
 4. The digital stereomicroscope apparatus of claim 1, wherein the image display comprise astereoscopic flat display panel that allows head movement of the humanoperator without losing a stereoscopic view of the area of interest. 5.The digital stereo microscope apparatus of claim 1, further comprising:at least one property sensor capturing data related to the area ofinterest and communicating the captured data to the processing system,wherein the processing system is configured to process the captured dataand display portions of the captured data on the image display inaddition to the magnified and augmented live stereo view.
 6. The digitalstereo microscope apparatus of claim 1, wherein the image displaycomprises a first image display, and further comprising: a second imagedisplay located apart from the display unit and communicatively coupledto the processing system, wherein the second image display is associatedwith a second human operator, wherein the second image display comprisesan independent magnified and augmented live stereo view from themagnified and augmented live stereo view of the first image display, andwherein the independent magnified and augmented live stereo view isformed by the processing system.
 7. The digital stereo microscopeapparatus of claim 1, further comprising: a display control controllingthe processing system responsive to the human operator for zoom, pan,and focus of the optical system and the image acquisition system via theprocessing system, wherein the zoom, pan, and focus is performedvirtually by the processing system.
 8. The digital stereo microscopeapparatus of claim 7, wherein the display control is configured to trackeye movements of the human operator and generate commands to configurethe optical system and the image acquisition system accordingly.
 9. Thedigital stereo microscope apparatus of claim 8, wherein the eye movementis approximated to the movement of the head and tracked.
 10. The digitalstereo microscope apparatus of claim 7, wherein ambient light intensityand direction of the light is controlled by the display control.
 11. Thedigital stereo microscope apparatus of claim 7, wherein the area ofinterest is remote from the human operator such that hands of the humanoperator and eyes of the human operator are virtually collocated. 12.The digital stereo microscope apparatus of claim 1, further comprising:cameras capturing a left eye view and a right eye view of a field ofview of the area of interest, wherein the processing system isconfigured to time synchronize the left eye view and the right eye viewfor presentation on the image display; and additional image sensingelements fixed to articulate arms for alternate views within the area ofinterest.
 13. The digital stereo microscope apparatus of claim 12,wherein the processing system is configured to utilize a sum of squareddifferences algorithm to identify pixel correspondence between thecameras; wherein the sum of squared differences algorithm selectivelyprocesses different regions of the area of interest based on whichregions are updated more frequently.
 14. The digital stereo microscopeapparatus of claim 1, further comprising: at least one flexible armadjustable separately from the display unit to the area of interest,wherein the at least one flexible arm comprises at least one componentof the image acquisition system.
 15. The digital stereo microscopeapparatus of claim 1, wherein the processing system comprises: propertydata processors that produce a visual representation of a property ofobjects in the area of interest as a time varying graphical view or as atime varying number display for display on the image display; controldata processors that process commands from the human operator forconfiguration and orientation of the optical system and the imageacquisition system; storage processors that store and retrieve data inan external storage system; and display processors that produce a stereoview of the area of interest and user interaction therewith and combinewith the visual representations of the property data processors andstorage processors to produce an augmented stereo view.
 16. The digitalstereo microscope apparatus of claim 1, wherein the processing system isconfigured to present the magnified and augmented live stereo view withsufficient disparity between the left and right eye view images suchthat the human operator sees a three dimensional representation throughstereopsis.
 17. A surgical operating system with a digital stereomicroscope, comprising: an articulated arm comprising a display mountingsystem; a display unit connected to the display mounting system; anoperating site over which the display unit is fixed and located suchthat a human operator places hands, tools, or a combination thereof inthe operating site and views a magnified and augmented live stereo viewof the operating site with eyes of the human operator having a viewsubstantially collocated and in a direct field and line of view with thehands, tools, or a combination thereof of the human operator in the areaof interest, wherein the human operator is able perform head movementwithout losing the live stereo view on the display unit; an imagedisplay located on a front side of the display unit relative to thehuman providing the magnified and augmented live stereo view; an opticalsystem and an image acquisition system located on a back side of thedisplay unit providing image capture and data capture of the operatingsite; and a processing system communicatively coupled to the imageacquisition system and the image display providing the magnified andaugmented live stereo view, in real-time, to the image display utilizingstereopsis based on the image capture and the data capture of theoperating site from the optical system and the image acquisition system.18. The surgical operating system of claim 17, wherein the opticalsystem and the image acquisition system comprises at least one objectivelens, at least one zoom lens, and at least one image sensor, and whereinthe optical system and the image acquisition system provide continuouszoom without an intermediate lens assembly; wherein the image displaycomprise a stereoscopic flat display panel that allows head movement ofthe human operator without losing a stereoscopic view of the area ofinterest.
 19. A method using a digital stereo microscope, comprising:positioning and adjusting a display unit above an operating site to afixed position; enabling the display unit, wherein the display unitcomprises an optical system and an image acquisition system located on aback side of the display unit adjacent to the operating site, andwherein the display unit comprises an image display on a front side ofthe display unit opposite to the operating site; providing image captureand data capture of the area of interest via the optical system and theimage acquisition system; processing the image capture and the datacapture via a processing system; positioning a user's hands, tools, or acombination thereof in the operating site while maintaining the user'seyes having a view in a collocated manner and in a direct field and lineof view looking at the image display, wherein the front side and theback side are relative to the user; and presenting a magnified andaugmented live stereo view of the operating site, in real-time, via theimage display utilizing stereopsis based on the processed image captureand the processed data capture of the area of interest from the opticalsystem and the image acquisition system, wherein the human operator isable perform head movement without losing the live stereo view on thedisplay unit.